Telepresence system, telepresence method, and video collection device

ABSTRACT

A telepresence system, a telepresence method, and a video collection device are disclosed. The telepresence system includes a video collection device, a video display device, an audio collection device, an audio player, and an audio and video communication device. The audio and video communication device transmits videos collected by the video collection device on a local end and audios collected by the audio collection device on the local end to a remote end through a network; the video display device and the audio player on the remote end play the images and audios respectively; and the video collection device is a panoramic camera. The technical solution under the present invention overcomes the poor effect of panoramic presence in the existing telepresence system, and improves the telepresence system in terms of depth presence, seamless display and eye contact.

CROSS-REFERENCE TO RELATED APPLICATION

This application is of U.S. Ser. No. 12/888,769, filed on Sep. 23, 2010,which is a continuation of International Application No.PCT/CN2009/071745, filed on May 12, 2009, all of which are herebyincorporated by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates to videoconference technologies, and inparticular, to a telepresence system, a telepresence method, and a videocollection device.

BACKGROUND

Telepresence is a videoconference system for implementing a virtualconference environment. The virtual conference environment aims to:reflect the personalization factors of the attendees sufficiently,simulate the real experience of the attendees as far as possible,improve the acceptability to the end users immensely, and improve theequipment use ratio, return on investment and user satisfaction.Compared with a traditional videoconference system, an idealTelepresence system brings more merits: images that simulate the size ofa real person; smooth motion; precise gestures; video, lighting, andaudio of a studio level; eye contact and communication like eye contactamong a large user group; immersive conference environment, which makesthe attendees feel as if they are on the same conference site;consistency of difference conference sites; and hidden cameras, whichreduce impact on the users.

In the process of implementing the present application, the inventorfinds that the existing telepresence system is defective in poorpanoramic presence effect. The existing telepresence system generallyuses multiple cameras and multiple large-screen monitors for collectingand displaying images. Each camera or monitor collects or displays oneor more persons on the local or remote site. In the existingtelepresence system, multiple cameras are used for photographing, anddisparity exists between the images photographed by different cameras.It is impossible to splice multiple images into panorama by laying outthe cameras. It is necessary to use the monitor rim to cover up thedefects of the images at the perspective joint of the cameras.Therefore, the existing telepresence system is unable to give pleasantpanoramic experience to the attendees. When the attendees are moving inthe area near the monitor rim, the image effect is even unacceptable.

Moreover, the existing telepresence need to improve the followingaspects:

1. Depth Presence

Most telepresence system still presents two-dimensional (2D) images.That is, the users see planar images only, and are unable to perceivethe depth information of the conference scene of the opposite party.

2. Seamless Display

The existing telepresence system generally uses multiple large-sizedflat televisions, either Liquid Crystal Display (LCD) or Plasma DisplayPanel (PDP), to present images in a combined way. In the adjacentdisplay area between two flat televisions, part of the images presentedin such a way is obstructed by the television rim, thus making itimpossible to give panoramic seamless experience to the attendees.

3. Eye Contact/Gaze Perception

Eye-to-eye contact is an important non-lingual communication mode. Eyecontact causes heart beats and blood pressure change physiologically,and improves the activity of the brain. Gaze perception provides manycommunication foundations such as feedback, dialog mode, and emotionexpression, and is a key means of perceiving the thoughts of theopposite party. The traditional videoconference system and the existingtelepresence system are unable to enable eye contact between users dueto disparity: Instinctively, the user looks at the opposite party on thescreen rather than the camera, but the camera is usually not located atthe center of the screen. Consequently, disparity exists between thepicture photographed by the camera and the picture faced by the user,and good eye contact is impossible.

SUMMARY OF THE INVENTION

The embodiments of the present invention provide an improvedtelepresence system, telepresence method, and video collection device topresent a good panoramic effect, and improve the depth presence,seamless display, and eye contact in the telepresence system.

The technical solution under the present invention includes thefollowing:

A telepresence system includes: a video collection device, configured tocollect images on a local end; an audio collection device, configured tocollect audios on the local end; a video display device, configured todisplay images from a remote end; an audio player, configured to playaudios from the remote end; an audio and video communication device,configured to transmits the images collected by the video collectiondevice on the local end and audios collected by the audio collectiondevice on the local end to the remote end through a network, the imagesand the audios are displayed and played respectively by a video displaydevice and an audio player on the remote end;

wherein the video collection device is a panoramic camera, and thesystem further comprises an image mosaics unit, configured to splicelow-resolution images photographed by the panoramic camera fromdifferent perspectives into a high-resolution panoramic image.

A telepresence method includes:

obtaining local panoramic images and audios, photographing imagesthrough a panoramic camera from different perspectives, and splicinglow-resolution images photographed by the panoramic camera fromdifferent perspectives into a high-resolution panoramic image through animage mosaics unit; and

transmitting local panoramic images and audios to a remote end through anetwork for displaying and playing.

A video collection device in a telepresence system is provided. Thetelepresence system further includes a video display device, an audiocollection device, an audio player, and an audio and video communicationdevice. The audio and video communication device transmits the imagescollected by the video collection device and the audios collected by theaudio collection device to the remote end through a network, and thevideo display device and the audio player on the remote end display andplay the images and audios. The video collection device is a panoramiccamera. An image mosaics unit splices the low-resolution imagesphotographed by the panoramic camera from different perspectives into ahigh-resolution panoramic image.

It can be seen from the above description that, the embodiments of thepresent invention are upgrade from the existing telepresence system. Theordinary camera can be replaced with a panoramic camera to photographthe panorama of the local conference room and provide a conferencepanorama for the opposite attendees. In this way, the telepresencesystem gives a good panoramic presence effect, and is compatible withthe existing telepresence system.

Preferably, an ordinary projection screen or holographic transparentprojection screen is employed to present the images seamlessly in anintegrated way, thus implementing seamless presence and overcoming thedefect brought by combination of multiple flat televisions.

Preferably, a holographic transparent projection screen and asemi-reflective semi-transparent mirror are employed to provide depthpresence for the attendees.

Preferably, through control of a synchronizing unit, the panoramiccamera is free from impact caused by the image projection of theprojector when photographing the local images, thus avoiding disparitycaused by inability of placing the camera in the line of sight of theuser and enabling the opposite attendee to enjoy the eye contact.Besides, the semi-reflective semi-transparent mirror or an opticalconduction component or a linear polarizer may be used to enable eyecontact.

Preferably, a special dark background is deployed, a backgroundprojector or background monitor is used, and a dark background isdeployed behind the user. In this way, the user image is separated fromthe background image, and the effect of depth presence is generated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is the first planform of a conference room layout in atelepresence system in a first embodiment of the present invention;

FIG. 2 is the first schematic diagram of a telepresence system in afirst embodiment of the present invention;

FIG. 3 is a holographic projection diagram of a telepresence system in afirst embodiment of the present invention;

FIG. 4 is schematic diagram of a panoramic camera in a telepresencesystem in a first embodiment of the present invention;

FIG. 5 is schematic diagram of a multi-reflective panoramic camera in atelepresence system in a first embodiment of the present invention;

FIG. 6 is a schematic diagram of a telepresence system in a firstembodiment of the present invention;

FIG. 7 is the second planform of a conference room layout in atelepresence system in a first embodiment of the present invention;

FIG. 8 is the second schematic diagram of a telepresence system in afirst embodiment of the present invention;

FIG. 9 is the third planform of a conference room layout in atelepresence system in a first embodiment of the present invention;

FIG. 10 is the third schematic diagram of a telepresence system in afirst embodiment of the present invention;

FIG. 11 is a planform of a conference room layout in a telepresencesystem in a second embodiment of the present invention;

FIG. 12 is the first schematic diagram of a telepresence system in asecond embodiment of the present invention;

FIG. 13 is the second schematic diagram of a telepresence system in asecond embodiment of the present invention;

FIG. 14 is a planform of a conference room layout in a telepresencesystem in a third embodiment of the present invention;

FIG. 15 is a schematic diagram of a telepresence system in the thirdembodiment of a present invention;

FIG. 16 is a schematic diagram of a telepresence system in a fourthembodiment of the present invention;

FIG. 17 is a schematic diagram of a telepresence system in a fifthembodiment of the present invention;

FIG. 18 is a schematic diagram of a telepresence system in a sixthembodiment of the present invention;

FIG. 19 is the first schematic diagram of a telepresence system in aseventh embodiment of the present invention;

FIG. 20 is the second schematic diagram of a telepresence system in aseventh embodiment of the present invention;

FIG. 21 is the third schematic diagram of a telepresence system in aseventh embodiment of the present invention;

FIG. 22 is the fourth schematic diagram of a telepresence system in aseventh embodiment of the present invention; and

FIG. 23 is a flowchart of a telepresence method in an embodiment of thepresent invention.

DETAILED DESCRIPTION

In order to make the technical solution, objectives, and merits of thepresent invention clearer, the following describes the embodiments ofthe present invention in more detail with reference to accompanyingdrawings and exemplary embodiments.

First, the first embodiment of the present invention is described below.

FIG. 1 is the first planform of a conference room layout in atelepresence system in the first embodiment of the present invention. InFIG. 1, the front wall 13 and the rear wall 14 of the site are an arc orplane, and a conference table 2 is set in the middle. A microphone array3 is installed on the conference table 2. In order to collect theconference audio data more effectively, the microphone array 3 may beplaced in the middle of the conference table 2. Multiple attendee seats1A, 1B, and 1C are placed on one side of the conference table 2. Theattendee seats face a projection screen 4. The projection screen 4 is anarc or plane shape (in the Figure, the projection screen is an arcshape), and makes up a front surface of camera bellows (dark enclosure)5. The camera bellows 5 holds a panoramic camera 7 (here the panoramiccamera includes three cameras, as shown in FIG. 4), an audiocommunication device 8, multiple projectors 9A, 9B, and 9C, and multiplespeakers. For example, five speakers 11A, 11B, 11C, 11D, and 11E make upa speaker array in the Figure. The inner wall 6 of the camera bellows 5opposite to the projection screen 4 is a backdrop decorated specially.The special decoration refers to decoration that produces a depth prompteffect and can hide cameras. The back of the attendee seat is a darkbackdrop 13 decorated specially. An auxiliary monitor 12 is placed onone side of the conference room. One or more object cameras 10A and 10Bare placed above the camera bellows 5 to make up one or morestereoscopic camera pairs. One stereoscopic camera pair is made up oftwo ordinary object cameras, which simulate the visual principles of ahuman and photograph the scene from the left and right perspectivessimultaneously to obtain a left image and a right image.

FIG. 2 is the first schematic diagram of the first embodiment of thepresent invention. FIG. 2 shows that the local telepresence site and theremote telepresence site have the same configuration, and areinterconnected through a network.

Preferably, in the first embodiment, the local projection screen A4 andthe remote projection screen B4 are holographic transparent projectionscreens. The holographic transparent projection screens are rearprojection screens based on a hologram technology, and are characterizedby holographic images. They display only the images from a specificperspective of the rear projection, but ignore the light rays from otherperspectives. The holographic screens generate very bright and cleardisplay effects, even if the environment light rays are very bright, andare transparent so that the audience can see the objects behind thescreen. Currently, some manufacturers like Woehburk, HoloPro, and Sax3Dmanufacture the holographic transparent projection screens. FIG. 3displays the basic principles of the holographic projection. As shown inFIG. 3, the projection rays that take on a α angle are scattered by theholographic transparent projection screen composed of holographicmaterials and transparent panels. In this way, the user can see theprojection content but cannot see the objects behind the projectioncontent area. However, the horizontal rays emitted by the object behindthe projection screen except the projection content are transmitted bythe projection screen. Through such projection screen area, the user cansee the object behind the projection screen.

In the first embodiment, an installation angle α exists between theprojection angle of the projector A9 in the camera bellows A5 and thehorizontal line. The projector A9 projects the image of the remoteattendee B15 to the local holographic transparent screen A4. Because thepanorama of the remote scene B is presented, the resolution of oneprojector may be not enough. The audio and video communication device A8splits the remote image into several parts, which are presented bymultiple projectors A9 (because the schematic diagram in FIG. 2 is aright view, the multiple projectors are overlapped and are not allpresented). In order to hide the projectors, the color of the projectorsA9 is preferably the same as the color of the camera bellows backgroundA6 behind.

In order to collect the panoramic image of the local A, a panoramiccamera A7 is installed in the camera bellows A5. FIG. 4(A) and FIG. 4(B)illustrate the basic principles of a solution to imaging of a panoramiccamera. The panoramic camera is based on the plane mirror reflectanceand virtual common optical center (panoramic camera in a virtual commonoptical center mode). The prismoid 1001 has three reflectance surfaces1002, 1003, and 1004. Such surfaces are plane mirrors, and three camerasC01, C02 and C03 are placed under the mirrors. The virtual commonoptical center is described below, taking one camera C02 as an example.As shown in FIG. 4(B), L02 is an incident ray, and R02 is a reflectiveray. The normal line 1006 is vertical to the reflective plane 1003, andthe angle between the normal line 1006 and the horizontal line 1010 isθ. The vertical distance from the reflection point to the actual opticalcenter 02 of the camera C02 is d. According to the light reflectanceprinciples, the camera photographs a virtual image, and the virtualimage has a virtual optical center 00. If the values of θ and d are setproperly, the virtual optical centers of the cameras C01, C02 and C03coincide, and three images that share an optical center are obtained.The three images are spliced to obtain an image which is seamlesslyspliced in any depth. In designing a panoramic camera, the location ofthe optical center of the camera is as low as practicable in order toobtain a better vertical eye-to-eye effect. If the geometric height ofthe camera is constant, such an effect can be accomplished by reducingthe horizontal distance between the camera and the reflection mirror.However, that distance is restricted by the size of the camera lens andviewfinder, and reduces the photographing perspective, as shown in FIG.4(C). Another solution to the panoramic camera is a multi-cameraaggregation model. Multiple images are photographed and spliceddigitally to obtain a panoramic image (panoramic camera in theaggregation mode), as shown in FIG. 4(D). Because the optical center isinside the camera, such a camera model is unable to share the opticalcenter by solely relying on the camera layout. Disparity exists in theoverlap of the images. Image processing technologies need to be appliedto achieve a good splicing effect. Another mode of the panoramic camerais to use a multi-camera array (panoramic camera in the multi-cameraarray mode), as shown in FIG. 4(E). The camera array may be in differentforms according to different scenes, for example, linear array, ringarray, or rectangular array. In the camera array, every camera has a lowresolution, and the intervals between adjacent cameras are small. Alarge photographing overlap exists. The image splicing technologysplices multiple camera images of low resolution into a high-resolutionpanoramic image. The basic principles of the image splicing algorithmare to estimate the internal parameters (such as focal length, principalpoint, and distortion) of multiple cameras and the inter-camera locationparameters (such as spin matrix, and translation vector); through theestimated parameters and the algorithm, the images of the multiplecameras are aligned, the overlap is eliminated, the edges are blended,and the disparity is eliminated to obtain a high-resolution panoramicimage.

The well-known image splicing technology is described below.

The basic principles of the image splicing algorithm are to estimate theinternal parameters of multiple cameras and the inter-camera locationparameters; through the estimated parameters and the algorithm, theimages of the multiple cameras are aligned, the overlap is eliminated,the edges are blended, and the disparity is eliminated to obtain ahigh-resolution panoramic image. According to the projection geometryprinciples, when a spatial 3D point is projected to a camera imagingplane, the transformation relation is:

x=K[R|t]X   (1)

$\begin{matrix}{K = \begin{bmatrix}f_{x} & s & u_{0} \\0 & f_{y} & v_{0} \\0 & 0 & 1\end{bmatrix}} & (2)\end{matrix}$

In the formula above, x is the homogeneous expression of planarcoordinates; X is the homogeneous expression of the world coordinatesystem; f_(x) and f^(y) are equivalent focal lengths in the horizontaland vertical directions respectively; s is a distortion coefficient ofthe image; and u₀,v₀ are principal point coordinates of the image. R isthe spin matrix of the camera, and t is the translation vector of thecamera. K is an internal parameter of the camera, and R and t areexternal parameters of the camera. For multiple images which haveoverlaps and are photographed by two cameras or photographed by onecamera in different locations, the imaging relation of a spatial pointin two images is:

x ₁=H₀₁ x ₀   (3)

H is a 3*3 matrix, whose freedom is 8. It represents the transformationrelation between two imaging planes, and is called a homography. For apure rotation camera system or a common optical center camera systemwhich involves only rotational motion, H may be expressed as:

H ₀₁ =K ₁ R ₁ R ₀ ⁻¹ K ₀ ⁻¹   (4)

Therefore, through a feature extraction algorithm such asScale-Invariant Feature

Transform (SIFT) algorithm, features are extracted in the overlap,multiple features are found, and a matching relation between features isset up. Multiple equation sets are created through (3), and thehomography H between two images is worked out through an iterativeoptimization algorithm. After the H is worked out, the two images can bespliced together through coordinate transformation, and the pixels inthe overlap are aligned. For the camera model that rotates in only thehorizontal direction, we can use cylindrical coordinate transformationto convert the planar coordinates into cylindrical coordinates. In thecylindrical coordinates, the pixels are aligned through imagetranslation. The transformation and inverse transformation of thecylindrical coordinates are:

$\begin{matrix}{{x^{\prime} = {s\; \tan^{- 1}\frac{x}{f}}}{x = {f\; \tan \frac{x^{\prime}}{s}}}} & (5) \\{{y^{\prime} = {s\frac{y}{\sqrt{x^{2} + f^{2}}}}}{y = {f\frac{y^{\prime}}{s}\sec \frac{x^{\prime}}{s}}}} & (6)\end{matrix}$

After the image is transformed according to the foregoing method, otherfactors need to be considered before an anticipated seamless panoramicimage is obtained. A major factor is disparity. The existing algorithmscan handle only the splicing in a certain depth of the image, namely,like the splicing on one plane. In theory, it is impossible to splicethe objects in other depths seamlessly through one transformation. Theobjects except those in this depth involve fringes. It is hard toeliminate the fringes through the image processing algorithm. A bettersolution is to minimize disparity through a common optical center cameramodel. Another factor is luminance/chroma difference between imagescaused by exposure/color difference between the cameras, especially atthe joint between two images. A simple solution is to perform Alphablending at the overlap of the joint, and a better solution is toperform Laplacian pyramid blending or gradient domain blending on thewhole image. After the relevant processing is finished, a betterpanoramic seamless image is obtained.

In order to obtain a better vertical eye-to-eye effect, the panoramiccamera A7 is preferably installed on a height approximately equivalentto the line of sight A100 of the attendee (see FIG. 2). The panoramiccamera A7 is made up of several ordinary color cameras. In order tophotograph the rapidly moving objects in the scene, the color camerasneed to be synchronized. Because the images obtained by multiple camerasmay be not suitable for splicing directly, the three channels of videoimages need to be spliced through an image splicing algorithm to obtaina seamless panoramic image. The multiple channels of video streamsoutput by the panoramic camera A7 may be transmitted directly to theaudio and video communication device A8 which splices the images.Alternatively, the panoramic camera A7 is connected directly to athird-party device (not illustrated in the Figure) for splicing theimages. After completion of splicing, the panoramic image is input tothe audio and video communication device A8. Alternatively, thepanoramic camera A7 splices the images, and inputs the spliced image tothe audio and video communication device A8 through one or more channelsvideo streams. The device capable of splicing images is called an imagemosaics unit herein. The principles of the image mosaics unit aredescribed above, and the connection relation between the image mosaicsunit and other units is described in the following text about FIG. 6. Asregards display, a single projector is unable to display the panoramicimage properly. Preferably, the panoramic image is split into severalparts, and each projector displays a part of the image. Because theprojectors differ in location, luminance and chroma, the split panoramicimage needs to be corrected geometrically, and the seam between adjacentimages needs to be eliminated through luminance/chroma blending. Such afunction is performed by an independent third-party device (notillustrated in the Figure), or integrated into the audio and videocommunication device A8. The device capable of image correcting/blendingis called a correcting/blending unit herein. For details, see thedescription about FIG. 6 later. In order to hide the panoramic cameraA7, the color of the camera is preferably the same as the color of thecamera bellows background A6 behind so that the camera is hardlynoticeable to the user.

In the Figure that illustrates the first embodiment, the panoramiccamera 7 is placed vertically, and the incident rays are reflected intocamera directly through a reflection mirror. In practice, the opticalpath of the incident ray is changed through repeated reflectance, andthe panoramic camera may be placed as required. FIG. 5 shows a solutionto placing the panoramic camera horizontally. A viewfinder 2 is addedabove the viewfinder 1 of the panoramic camera. Therefore, thehorizontally transmitted rays are changed to vertically transmittedrays, and the location of the camera may be changed. Because there aremultiple cameras, a proper reflectance plane needs to be designed on theupper side of each camera.

In order to prevent the local image A photographed by the panoramiccamera A7 from being affected by the image projected by the projectorA9, preferably, the first embodiment of the present invention uses atime division method to coordinate the collection of the camera A7 withthe projection of the projector A9. According to the time divisionmethod, the working modes of the system are categorized into two modes:display mode and collection mode. In the display mode, the projector A9projects the image of the remote end B to a transparent projectionscreen A4. At this time, the panoramic camera A7 is inactive and doesnot collect signals; in the collection mode, the projector A9 isinactive and does not project images, and the panoramic camera A7photographs the scene through a transparent projection screen A4. Inorder to coordinate the camera A7 and the projector A9, a specialsynchronizing unit A16 is required to output synchronization signals tothe panoramic camera A7 and the projector A9, and control the workingmode of the two devices. For example, the synchronizing unit A16controls the panoramic camera A7 to collect signals in the verticalflyback interval between two frames/scenes of images of the projectorA9. At this time, however, the exposure time of the panoramic camera A7is shorter, and the luminance of the image is lower. In order to solvesuch problems, the camera of a shorter exposure time may be applied, orthe refresh rate of the projector may be reduced.

As described above, the panoramic camera B7 obtains a panoramic image ofthe scene of the remote user B15, and the image is presented on aprojection screen A4 locally. Therefore, the local user A15 feels as ifthe user is surrounded by the remote scene, and perceives a seamlesslydisplayed panoramic image, without the noticeable impression of speakingto a screen. The user's sense of immersion is enhanced. Moreover, theimage of the remote user B15 is presented on a local transparentprojection screen A4, and the surroundings of the remote user B15 are adark background, and will not be imaged on the transparent projectionscreen A4. Therefore, the local user A15 can see the background A6 ofthe camera bellows AS through such a part. A physical distance existsbetween the transparent projection screen A4 and the background A6 ofthe camera bellows A5, and the background A6 of the camera bellows AS isdecorated specially, which brings a depth illusion to the user.Therefore, the local user A15 perceives the depth of the image of theremote user B15. Moreover, as controlled by the synchronizing unit A16,the panoramic camera A7 is free from impact of the projection of theprojector A9 when photographing the local image A. Likewise, thepanoramic camera B7 is free from impact of the projection of theprojector B9 when photographing the remote image B. Therefore, thecameras may be placed behind the projection screen center along the lineof sight of the attendee, thus avoiding vertical disparity and enablingthe opposite attendee to enjoy eye contact.

The synchronizing unit A16/B16 enables face-to-face video communication.In addition, the telepresence system involves remote collaborationtasks, for example, two design teams need to see the design prototype.The existing telepresence system supports 2D videos only, and the useris unable to see an object that takes on a depth sense. The solution putforward in this embodiment may use a stereoscopic camera as an objectcamera to accomplish 3D videos when presenting the object. As shown inFIG. 2, the stereoscopic camera B10 on the remote end B collects the 3Dimage information of an object to be presented, for example, “left eyeimage+right eye image”, or “left eye image+depth image”, and inputs theinformation to the audio and video communication device B8 capable of 3Dvideo coding. The audio and video communication device B8 processes the3D image, encodes the image, and sends it to the audio and videocommunication device A8 on the local end A. The audio and videocommunication device A8 on the local end A decodes and presents the 3Dvideo code streams. If a 3D video presence device exists locally, thevideo is presented as 3D video; otherwise, the video is presented as 2Dvideo. For example, if the auxiliary monitor A12 on the local end A is a3D monitor, the audio and video communication device A8 outputs a videoof the 3D format to the A12 for displaying. If the local auxiliarymonitor A12 is an ordinary 2D monitor, the audio and video communicationdevice A8 outputs a video of the 2D format to the A12 for displaying.The 3D video presence devices include 3D glasses, automatic stereoscopicmonitor, and multi-perspective 3D monitor.

In order to give pleasant immersive audio experience, a microphone arrayA3 and a speaker array A11 are preferred to present audios. Thebackgrounds of the microphone array and the speaker array are outlinedbelow. In a telepresence system, the reproduction of the audio includesthe reproduction of acoustics and the reproduction of the stereoscopicsense. The reproduction of acoustics can be accomplished through anefficient wide-frequency compression algorithm. The stereoscopic sensebrings impression of locations and directions, enhances the impressionof being in the same room, makes the voice more understandable, andmakes the speechmaker quickly identifiable. The first embodiment of thepresent invention uses multiple microphones or microphone arrays tocollect audios, and uses multiple speakers or speaker arrays to presentaudios, thus improving the effect of reproduction of the stereoscopicsense of the sound. The microphone array is a system of unidirectionalmicrophones distributed in a certain geometrical structure. Atraditional directional microphone generally collects only one channelof signals, but a microphone array system collects multiple channels ofsignals. Because the microphones are located differently, the collecteddata differs chronologically or spatially. Through the signal processingtechnology, the required information may be extracted from multiplechannels of signals. The microphone array system is characterized byspace selection. The beams generated by the microphone aim at the soundsource, and suppress the sound of other speechmakers and the environmentnoise, thus giving high-quality sound source signals. Currently, soundsource positioning is a main application scenario of the microphonearray. The positioning based on a microphone array is to determine thespatial location of the sound source by using the microphonesdistributed in a specific geometrical layout. If the sound sourcepositioning algorithm based on a microphone array comes in three types:controllable beam formation technology based on the maximum outputpower, direction determining technology based on high-resolutionspectrum estimation, and Time Difference Of Arrival (TDOA)-basedtechnology. The first method is to filter the voice signals received bythe microphone array, sum up the weighted value of the voice signals,and then control the microphone to point toward the direction that makesthe maximum output power of the beam. The second method is to determinethe direction angle by working out the relevant matrix between themicrophone signals, and determine the location of the sound source. Thethird method is to work out the time differences of the sound arrivingat the microphones in different locations, use such time differences towork out the distance differences of the sound arriving at themicrophones in different locations, and then determine the location ofthe sound source through search or geometrical knowledge. The speakerarray can rebuild and reproduce the sound field according to the inputaudio signals and location information. The speaker array can combinemultiple sound field units in a certain way to amplify sound. Comparedwith a single speaker which radiates sound directly, the speaker arrayincreases the sound power, increases the sound radiation effect in thecase of low frequency, improves the directionality and unevenness of thesound field, and improves the voice clarity in a reverberationenvironment. The speaker array can be based on a wavefront synthesistechnology.

The microphone array A3 may be a linear microphone array or a circularmicrophone array, and is placed on the table or suspended from theceiling. The speaker array A11 includes many speakers, which areintegrated with the camera bellows A5. The distribution direction of thespeakers is the same as the distribution direction of the remoteattendees displayed on the projection screen A5. In the conferenceprocess, through the microphone array B3 on the remote end B, the audioand video communication device B8 can detect the location of a speakingattendee B15, and transmit the audio signals on the remote end B and thelocation information of the attendee B15 to the audio and videocommunication device A8 on the local end A. The speaker array A11 canrebuild and reproduce the sound field according to the input audiosignals and location information. In this way, the local user A15 feelsthat the sound of the remote user B15 is uttered from the B15 locationon the screen, and obtains the experience like a face-to-face talk.

FIG. 6 displays the system working flowchart of the first embodiment,where the remote end B transmits audio and video information to local A.

On the remote end B, the panoramic camera B7 (composed of multiplecameras) collects images of the scenes in different perspectivessynchronously under control of the synchronizing unit B16. The multiplecollected images are sent to the image mosaics unit for splicing into apanoramic image of a remote scene B. This panoramic image is output fromthe image mosaics unit, processed and further output to the video codingunit 1 for coding, and then transmitted through a packet-switched domainnetwork in the form of packet code streams. It is worthy of attentionthat the resolution of the spliced images may be very high, and onevideo coding unit 1 may be unable to encode the spliced image in realtime. The spliced image may need to be split into several parts, andoutput to multiple video coders synchronously for coding. After coding,the image may form one or more code streams, which are transmittedthrough the packet-switched domain network. Due to distributed codingand the delay and jitters caused in the network transmission, the codestreams may be out of sync. Therefore, the code streams may need to belabeled (for example, through timestamps). At the decoder, the codestreams are synchronized according to the labels. Likewise, themicrophone array on the remote end B collects the audio signals of thescene, and encodes the signals through an audio coding unit to formencoded audio code streams, which are transmitted through the network inthe form of data packets. In order to prevent loss of synchronizationbetween the audio and the video, it is better to synchronize for theaudio data and the video data. The synchronization of audio data andvideo data is a prior art in the audio and video field, and is notrepeated herein any further. Because the remote end B uses a microphonearray to collect the audio signals, the audio positioning algorithm isalso capable of calculating out the location information of a speakingattendee. The location information may be output to the local endthrough the network. Apart from the panoramic camera B7, one or moreobject cameras B10 photograph the objects that need to be presented inthe scene. If there are multiple object cameras B10, they may make up astereoscopic camera for obtaining the 3D image of the scene. In thiscase, a synchronizing unit B16 also exists between the cameras forsynchronizing the collection. One or more video streams of the objectcamera B10 are input to the video coding unit 2 for coding. The videocoding unit 2 supports 2D/3D video coding formats, and the encoded codestream data is transmitted through the packet-switched domain network.

On the local end A, the video coding unit 1 receives the panoramic videocode streams from the remote end B for decoding. Because the resolutionof the panoramic image may be very high, one video decoding unit 1 isunable to finish decoding the image, and multiple video decoding unitsmay need to work concurrently. In the decoding, the sequence of playingthe video image frames needs to be determined according to thesynchronization label in the code streams. After decoding, the image(s)may be a complete panoramic image or multiple split images. For acomplete panoramic image, the image needs to be split into multipleimages, which are output to multiple projectors A9 synchronously. Themultiple projectors A9 presents the images on the projection screen A4seamlessly. Before the projector presents the image, because theprojectors differ in location, luminance, and chroma, the image ispreferably corrected geometrically through a correcting/blending unit,and the seam between adjacent images needs to be eliminated throughluminance/chroma blending. The audio data code streams are decoded bythe audio decoding unit into audio data signals, which are output to thespeaker array. According to the location information of the attendee onthe remote end B, one or more speakers in the speaker array closest tothe remote attendee displayed on the projection screen A4 may beselected to to present the audio of the remote attendee. The video codestreams of the object camera B10 on the opposite side B are decoded bythe video decoding unit 2, and presented by the auxiliary monitor A12.If the auxiliary monitor A12 supports 3D videos, the videos arepresented as 3D videos; if the auxiliary monitor A12 supports 2D videosonly, the videos are presented as 2D videos. Therefore, a complete audioand video communication device A8 or B8 includes: an image mosaics unit,video coding units 1 and 2, an audio coding unit, video decoding units 1and 2, and an audio decoding unit.

FIG. 7 is the second planform of a conference room layout in atelepresence system in the first embodiment of the present invention,and FIG. 8 is the second schematic diagram of a telepresence system inthe first embodiment of the present invention. The solution is based onthe front projection technology. The projector A9 may be placed in frontof the projection screen A4, and suspended above the conference table A2(as shown in the Figure) or below the conference table for projection.The solution is superior because the rays emitted by the projector A9causes no interference to the user.

FIG. 9 is the third planform of a conference room layout in a simplifiedsolution in the first embodiment of the present invention. FIG. 10 isthe third schematic diagram of a conference room layout in a simplifiedsolution in the first embodiment of the present invention. In thesimplified solution, the panoramic camera A7 is placed above theprojection screen A4 to obtain a panoramic image of the scene. In thiscase, no synchronizing unit A16 is required for synchronizing thecollection of the panoramic camera A7 and the projection of theprojector A9, thus simplifying the design and reducing the cost of thepanoramic camera A7 and the projector A9. However, because the panoramiccamera 7 is not placed on the line of sight of the user A15, thevertical eye-to-eye effect is deteriorated. Generally, if the verticalperspective difference is less than seven degrees (<7°), the verticaleye-to-eye effect is acceptable. In order to reduce the verticalperspective, the viewfinder of the panoramic camera A7 may be placedbelow the color camera.

In the simplified solution shown in FIG. 9 and FIG. 10, the projectionscreen A4 may be a holographic transparent projection screen or anordinary projection screen. With a holographic projection screen, theuser can perceive the depth of the scene; with the ordinary rearprojector, it is impossible to present the depth sense of the scene, andthe backdrop A13 behind the user and the backdrop A6 of the camerabellows A5 do not need to be decorated specially.

It can be seen from the above description that, in the three solutionsin the first embodiment, a panoramic camera (A7 or B7) is employed, thepanorama of the local conference room can be photographed, and theopposite attendee can obtain a conference panorama, thus bringing a goodeffect of presenting the panorama in the telepresence system.

No matter whether the projection screen A4 or B4 is an ordinaryprojection screen or a holographic transparent projection screen, theprojection screen presents the images in an integrated way, thusimplementing seamless presence and overcoming the defect brought by thetelevision flanges when multiple flat televisions are combined.

Preferably, the projection screen A4 or B4 is a holographic transparentprojection screen, which provides depth presence for the attendees.

Preferably, through control of a synchronizing unit A16 or B16, thepanoramic camera A7 or B7 is free from impact caused by the imageprojection of the projector A9 or B9 when photographing the images onthe local end A or remote end B, thus avoiding vertical perspectives andenabling the opposite attendee to enjoy the eye contact. The panoramiccamera A7 or B7 may also be placed above the projection screen A4 or B4to obtain a panoramic image of the scene. In this case, thesynchronizing unit A16 or B16 is omissible, thus simplifying the designand reducing the cost. In order to reduce the vertical perspectives, theviewfinder of the panoramic camera A7 or B7 may be placed below thecolor camera.

Preferably, the projector A9 or B9 is set in the camera bellows A5 or B5so that the projector A9 is free from impact of environmental light. Theprojector A9 or B9 may be placed in front of the projection screen A4 orB4 through a front projection technology, or suspended above theconference table or below the conference table. In this way, the rays ofthe projector A9 or B9 cause no interference to the user in theholographic transparent projection.

The second embodiment of the present invention is described below.

FIG. 11 is a planform of a conference room layout in a telepresencesystem in the second embodiment of the present invention. Correspondingto FIG. 11, FIG. 12 is the first schematic diagram of a telepresencesystem in the second embodiment of the present invention. In the secondembodiment, the camera bellows are omitted, and the projection screen 4is directly placed opposite to the seats 1A, 1B, and 1C. In actualdesign, the projection screen may be designed as an elevation structure.The projection screen 4 is elevated up or hidden in the table 2 when itis idle, and is lowered or comes up from the table 2 when it works. Inthis way, the table 2 can be used for ordinary conferences when novideoconference is underway. Because the projection screen 4 is verythin, the speaker may be a very thin flat panel speaker placed below theprojection screen 4; or a vibrating module is fitted onto the screendirectly so that the screen becomes a speaker. In order to accomplish agood vertical eye-to-eye effect, the panoramic camera 7 may be hiddeninto the background wall 14 behind the projection screen 4. Throughspecial decoration, the panoramic camera 7 and the background wall 14are blended together. The projector 9 needs to be synchronized with thepanoramic camera 7 through a synchronizing unit A16 in a way describedin the first embodiment. The projector 9 in this embodiment may employ afront projection mode (shown in FIG. 11) or rear projection mode.

Another design solution in the second embodiment 2 is to put thepanoramic camera 7 above the projection screen A. FIG. 13 is the secondschematic diagram of a telepresence system in the second embodiment ofthe present invention. In practice, a supporting flange of a specificthickness is designed, the projection screen A4 is embedded into theflange, and the panoramic camera A7 is placed in front of the flange orembedded into the flange. In the case, no synchronizing unit is requiredfor synchronizing collection of the panoramic camera A7 and projectionof the projector A9. However, because the panoramic camera 7 is notplaced on the line of sight of the user A15, the vertical eye-to-eyeeffect is deteriorated. Generally, if the vertical perspectivedifference is less than seven degrees (<7°), the vertical eye-to-eyeeffect is acceptable. In order to reduce the vertical perspective, theviewfinder of the panoramic camera A7 may be placed below the colorcamera.

In the second embodiment, as shown in FIG. 11, because the projectionscreen 4 is directly placed opposite to the seat, the projection screencan be retracted when it is idle. Therefore, this solution is compatiblewith the traditional conference. That is, the relevant devices can behidden when no telepresence conference is underway, and the conferenceroom is available for ordinary conferences.

The third embodiment of the present invention is described below.

FIG. 14 is a planform of a conference room layout in a telepresencesystem in the third embodiment of the present invention. FIG. 15 is aschematic diagram of a telepresence system in the third embodiment ofthe present invention. The third embodiment further simplifies thesolution. Instead of a projection screen, large-sized televisions 30A,30B, and 30C are spliced into a display system. The televisions may beLCD television, PDP television, or Digital Light Processing (DLP) rearprojection television. The panoramic camera A7 is placed above themonitor to photograph the local scene. Because the television presenceis unable to render the depth effect, the backdrop A13 behind the userand the backdrop A6 of the camera bellows need no special decoration.

It can be seen from the above description that, the third embodiment isan upgrade on the basis of the existing telepresence system. It gives agood effect of panoramic presence by only replacing the ordinary camerawith a panoramic camera.

The fourth embodiment of the present invention is described below.

FIG. 16 shows a telepresence system in the fourth embodiment of thepresent invention. A semi-reflective semi-transparent mirror is used inthis embodiment to present depth. The holographic transparent projectionscreen is replaced with a semi-reflective semi-transparent mirror A21.The semi-reflective semi-transparent mirror is installed in front of thecamera bellows. The projection screen A22 is sideways above thesemi-reflective semi-transparent mirror, and a certain angle existsbetween the projection screen and the semi-reflective semi-transparentmirror. The projection screen A22 forms images by means of rearprojection. The images projected by the projector A9 are reflected bythe reflection mirror A23, and then change to images on the projectionscreen A22. The semi-reflective semi-transparent mirror A21 makes theimages at A22 into virtual images A101, and makes the local A15 see theimages at a certain depth. The panoramic camera A7 collects images ofthe user through the semi-reflective semi-transparent mirror A21, and isblended with the background A6 of the camera bellows A5. A darkbackground behind the remote user enables the local background A6decorated specially to be visible to the local user A15 through the darkarea except the bright area where the body of the remote user issituated. Because a physical distance exists between the image A101 ofthe remote user visible to the local user and the local background A6,it looks as if the image of the remote user in the eye of the local useris in front of the background. It is to be noticed that the reflectionmirror A23 is optional. When the reflection mirror A23 is omitted, theprojector A9 may employ the front projection solution.

It can be seen from the above description that, the fourth embodimentaccomplishes panoramic presence through a panoramic camera A7, andaccomplishes depth presence and eye contact through a semi-reflectivesemi-transparent mirror A21 on the basis of realizing seamless displaythrough projection.

The fifth embodiment of the present invention is described below.

FIG. 17 is a schematic diagram of a telepresence system in the fifthembodiment of the present invention. This embodiment employs atransparent optical conduction component A25, which has an opticalredirection area A25 for inputting images. As a waveguide device, A24transmits the image of the attendee A15 to the panoramic camera A7 atthe bottom of the camera bellows A5 solely through inner reflection. Asshown in the Figure, the incident ray A102 is reflected repeatedlybetween two inner surfaces A26 and A27 of the light guide component, andis finally radiated as an emergent ray A103 at the bottom of the camerabellows A5 and collected by the panoramic camera A7. The opticalconduction component is placed in front of the projection screen A4, andthe input area A25 covers the surface of the projection screen A4. TheA25 needs to be transparent enough to prevent causing interference tothe user A15. The inner surfaces A26 and A27 of the input area may beaccomplished through holographically derived grating. The component is atransparent panel made up of glass or plastic.

It can be seen from the above description that, the fifth embodimentaccomplishes panoramic presence through a panoramic camera A7,accomplishes seamless display through a projection screen A4, andaccomplishes eye contact through an optical conduction component A25.Preferably, the projection screen A4 is a holographic transparentprojection screen capable of presenting depth.

The sixth embodiment of the present invention is described below.

This embodiment accomplishes a panoramic telepresence system thatsupports the vertical eye-to-eye effect through a polarizer.

The principles of the well-known polarized light are outlined below.

Light waves are transverse waves. That is, the vibration direction ofthe light wave vector is vertical to the propagation direction of thelight. Generally, for the light wave emitted from a light source, thevibration of the light wave vector takes on an irregular trend in thedirection vertical to the light propagation direction. Averagely, in alldirections in the space, the distribution of light wave vectors isdeemed as sharing equal chances. Their sum is symmetrical with that inthe light propagation direction. That is, the light vector ischaracterized axial symmetry, even distribution, and equal amplitude ofvibration in all directions. Such light is called natural light.Polarized light refers to the light wave whose light vector vibrationdirection does not change or changes regularly. Depending on the nature,polarized light is categorized into planar polarized light (linearpolarized light), circular polarized light, elliptical polarized light,and partially polarized light. If the vibration direction of theelectric vector of the light wave is limited to a definite plane, thepolarized light is called planar polarized light; and, if the orbit is astraight line in the propagation process, the polarized light is calledlinear polarized light. If the electric vector of the light wave changesregularly with time, namely, the end orbit of the electric vector isvertically a straight line in the propagation process, the polarizedlight is called linear polarized light. If the electric vector of thelight wave changes regularly with time, namely, the end orbit of theelectric vector is circular or elliptical on the plane vertical to thepropagation direction, the polarized light is called circular orelliptical polarized light. If the vibration of the electric vector ofthe light wave is relatively dominant only in a specific direction inthe propagation process, the polarized light is called a partiallypolarized light. A polarizer is a thin film made manually. Crystalparticles which absorb selectively are arranged in the transparentlayers regularly in a special way to form the polarizer. The polarizeris permeable to the light in a certain vibration direction of theelectric vector (this direction is called a polarization direction), butabsorbs the vertically vibrating light, namely, the polarizer takes ondichroism.

FIG. 18 is a schematic diagram of a telepresence system in the sixthembodiment of the present invention. In this embodiment, a linearpolarizer is placed in front of the lens of the projector A9 and thepanoramic camera A7. The polarization angle of the linear polarizer ofthe projector A9 is different from the polarization angle of the linearpolarizer of the panoramic camera A7. (The polarization direction of thepolarizer in front of the panoramic camera is different from thepolarization direction of the light projected by the projector. That isbecause, according to the principles of polarized light, the lightprojected by the projector in this case is unable to enter the camerathrough the polarizer in front of the panoramic camera.) In idealconditions, the difference is 90 degrees. That is, the polarizationdirection of the projector A9 is vertical to the polarization directionof the panoramic camera A7. For example, the polarization direction ofthe projector A9 is vertical, but the polarization direction of thepanoramic camera A7 is horizontal. A projection screen A4 made fromspecial materials may be used instead of the polarizer in front of thecamera. The projection screen is a semitransparent screen interwovenfrom the polarizer material A41 of the panoramic camera A7 and othermaterials A42. In this way, the input circular polarized light of thescene changes to horizontal linear polarized light after passing throughthe projection screen A4, and can be collected by the panoramic cameraA7; the light projected by the projector A9 is vertical linear polarizedlight and is unable to pass through the horizontal linear polarizer ofthe camera A7, and thus is not collectible by the camera A7. In thisway, the photographing of the panoramic camera A7 and the projection ofthe projector A9 generate no interference.

It can be seen from the above description that, the sixth embodimentaccomplishes panoramic presence through a panoramic camera A7,accomplishes seamless display through a projection screen A4, andaccomplishes eye contact through the polarizer added in front of thecamera and the projector.

The seventh embodiment of the present invention is described below.

The seventh embodiment aims to solve the layout of the dark backgroundbehind the user in the preceding solutions. In the preceding solution,in order to present depth, the background behind the user needs to befixed as a dark background, for example, a black curtain or blackpainted wall. Such a background may be unacceptable to the user in someconference rooms. For example, the user feels that the dark backgroundis not harmonized with the decoration design of the conference room.

FIG. 19 is the first schematic diagram of a telepresence system in theseventh embodiment of the present invention. A background projector A50is used to project the user background to be displayed onto a pure blackprojection curtain A13. The background projector A50 is connected withthe synchronizing unit A16. The synchronizing unit A16 coordinates thecollection of the panoramic camera A7 and the projection of thebackground projector A50. According to the time division method, theworking modes of such a system are categorized into two modes:background projection mode and camera collection mode. In the backgroundprojection mode, the background projector A50 projects the background toa black curtain A13. At this time, the panoramic camera A7 is inactiveand does not collect signals. In the camera collection mode, thebackground projector A50 is inactive and does not project images, andthe panoramic camera A7 photographs the scene. In this case, thebackground of the user A15 photographed by the panoramic camera A7 is adark background. The local background seen by the user is not black, butis an image projected by the background projector A50 only if the shiftis fast enough. The image is replaceable, and can be in harmony with thedecoration of the conference room.

FIG. 20 is the second schematic diagram of a telepresence system in theseventh embodiment of the present invention. In FIG. 20, a linearpolarizer is added in front of the background projector A50 so that thelight projected to the background wall is linear polarized light.Another linear polarizer is added in front of the panoramic camera A7,and the polarization angle of this polarizer is vertical to thepolarization angle of the polarizer in front of the background projectorA50. In this way, the background light projected by the backgroundprojector A50 is not collectible by the panoramic camera A7, and theconference room lamp light reflected by a foreground person iscircularly polarized, and can be photographed by the camera A7.Therefore, the background behind the person is black in the photographedimage, thus solving the dark background problem.

FIG. 21 is the third schematic diagram of a telepresence system in theseventh embodiment of the present invention. In FIG. 21, a large-sizedbackground monitor A51 is applied behind the user to display thebackground image. The background monitor A51 is connected with thesynchronizing unit A16. The synchronizing unit A16 coordinates thecollection of the panoramic camera A7 and the display of the backgroundmonitor A51. According to the time division method, the working modes ofthe system are categorized into two modes: background display mode andcamera collection mode. In the background display mode, the backgroundmonitor A51 displays the normal image. At this time, the panoramiccamera A7 is inactive and does not collect signals. In the cameracollection mode, the background monitor A51 displays a pure blackbackground image, and the panoramic camera A7 photographs the scene. Inthis case, the background of the user A15 photographed by the panoramiccamera A7 is a dark background. The image seen by the user and displayedby A51 is not black only if the shift is fast enough.

FIG. 22 is the fourth schematic diagram of a telepresence system in theseventh embodiment of the present invention. In FIG. 22, a largebackground monitor A51 is placed behind the person, a linear polarizeris added in front of the background monitor A51, and another linearpolarizer is added in front of the panoramic camera A7. If thebackground monitor A51 is an LCD monitor, the background light emittedby the LCD is polarized light. Therefore, only one linear polarizerneeds to be added in front of the panoramic camera A7, and thepolarization angle of the linear polarizer is vertical to thepolarization angle of the background polarized light emitted by thebackground monitor A51. In this way, the background light of thebackground monitor A51 is not collectible by the panoramic camera A7,and the conference room lamp light reflected by the foreground person iscircularly polarized, and can be photographed by the panoramic cameraA7. Therefore, the background behind the person is black in thephotographed image, thus solving the dark background problem.

It can be seen from the above description that, in the seventhembodiment, a background projector A50 or background monitor A51projects the user background to be displayed to a black projectioncurtain A13, thus solving the dark background layout behind the user.The seventh embodiment may be combined with embodiments 1-6.

In conclusion, this embodiment is an upgrade from the existingtelepresence system. The ordinary camera can be replaced with apanoramic camera to photograph the panorama of the local conference roomand provide a conference panorama for the opposite attendee. In thisway, the telepresence system gives a good panoramic presence effect, andis compatible with the existing telepresence system.

Preferably, an ordinary projection screen or holographic transparentprojection screen is employed to present the panoramic images in anintegrated way, thus implementing seamless presence and overcoming thedefect brought by combination of multiple flat televisions.

Preferably, a holographic transparent projection screen and asemi-reflective semi-transparent mirror are employed to provide depthpresence for the attendees.

Preferably, through control of a synchronizing unit, the panoramiccamera is free from impact caused by the image projection of theprojector when photographing the local images, thus avoiding disparitycaused by inability of placing the camera in the line of sight of theuser and enabling the opposite attendee to enjoy the eye contact.Besides, the semi-reflective semi-transparent mirror or an opticaltransmission component or a linear polarizer may be used to enable eyecontact.

Preferably, a special dark background is deployed, a backgroundprojector or background monitor is used, and a dark background isdeployed behind the user. In this way, the user image is separated fromthe background image, and the effect of depth presence is generated.

Moreover, a remote presence method is provided in an embodiment of thepresent invention. As shown in FIG. 23, the method includes thefollowing steps:

S2301: Obtain local panoramic images and audios, photograph imagesthrough a panoramic camera from different perspectives, and splicelow-resolution images photographed by the panoramic camera fromdifferent perspectives into a high-resolution panoramic image through animage mosaics unit.

S2302: Transmit local panoramic images and audios to a remote endthrough a network for displaying and playing.

The panoramic camera photographs the scene in any of these modes:virtual common optical center of planar reflection mirrors, convergentmulti-camera mode, and dense camera array mode.

Preferably, the images and audios are collected alternately in timeorder. Preferably, the method further includes: collecting the local 3Dvideos through a stereoscopic camera, transmitting the videos to theremote end through a network, and displaying the videos through anauxiliary display device. Preferably, before displaying the panoramicimage, the method further includes: performing geometrical correctionand edge blending for the panoramic image. Preferably, the methodfurther includes: receiving location information of the remote attendee,and rebuilding and reproducing the sound field for the received audiosaccording to the location information. Preferably, the method furtherincludes: synchronizing for the locally obtained audio data and videodata.

A video collection device in a telepresence system is provided in anembodiment of the present invention.

The video collection device works together with the video displaydevice, audio collection device, audio player, and audio and videocommunication device in the telepresence system. The audio and videocommunication device transmits the images collected by the local videocollection device and the audios collected by the local audio collectiondevice to the remote end through the network. The video display deviceand the audio player on the remote end display and play the images andaudios respectively. Compared with the prior art, the video collectiondevice in the embodiments of the present invention is a panoramiccamera, and an image mosaics unit is used to splice the low-resolutionimages photographed by the panoramic camera from different perspectivesinto a high-resolution panoramic image.

The image mosaics unit is an independent device, or a part of thepanoramic camera, or a part of the audio and video communication device.The panoramic camera photographs the scene in any one of these modes:virtual common optical center of planar reflection mirrors, convergentmulti-camera mode, and dense camera array mode.

Described above are merely some exemplary embodiments of the presentinvention, but not intended to limit the scope of the present invention.Any modification, equivalent replacement, and improvement made withoutdeparting from the spirit and principle of the present invention fallwithin the scope of the present invention.

What is claimed is:
 1. A telepresence system, comprising: a videocollection device, configured to collect images on a local end; an audiocollection device, configured to collect audios on the local end; avideo display device, configured to display images from a remote end; anaudio player, configured to play audios from the remote end; an audioand video communication device, configured to transmits the imagescollected by the video collection device on the local end and audioscollected by the audio collection device on the local end to the remoteend through a network, wherein the video display device comprises aprojection screen and at least one projector configured to project theimage of the remote end to the projection screen; a background disposedbehind the projection screen, wherein the projection screen is asemi-transparent projection screen or a transparent projection screen,so that the background is visible through the projection screen.
 2. Thesystem of claim 1, wherein the video collection device is a panoramiccamera, and the system further comprises an image mosaics unit which isconfigured to splice narrow images photographed by the panoramic camerafrom different perspectives into a broad panoramic image.
 3. The systemof claim 2, wherein: the transparent projection screen is a holographictransparent projection screen, and the panoramic camera is configured tocapture the images transmitted via the holographic transparentprojection screen.
 4. The system of claim 3, wherein: the system furthercomprises a synchronizing unit, configured to output synchronizationsignals to control the panoramic camera and the projector to workalternately.
 5. The system of claim 2, wherein, the system furthercomprises a transparent optical conduction component, the transparentoptical conduction component has an optical redirection area coveringthe projection screen; and the panoramic camera is configured tophotograph the images reflected by inner surface of the opticalredirection area.
 6. The system of claim 5, wherein the inner surface ofthe optical redirection area adopts holographically derived grating. 7.The system of claim 2, wherein: the video display device comprises theproject screen and at least one projector, and the system furthercomprises a linear polarizer which is placed in front of the projectorand a lens of the panoramic camera; and a polarization angle of apolarizer of the projector is different from a polarization angle of apolarizer of the panoramic camera.
 8. The system of claim 1, wherein theaudio player is at least one speaker array, the audio and videocommunication device is further configured to receive remote audios andremote attendee location information transmitted by the audio and videocommunication device on the remote end, wherein the location informationis detected by the audio and video communication device on the remoteend through the audio collection device on the remote end; the speakerarray is configured to select one or more speakers closest to the remoteattendee displayed on the projection screen to reproduce a sound fieldfor received audios.
 9. The system of claim 1, wherein the audiocollection device is at least one microphone array, the microphone arrayis configured to collect audios on the local end, calculate out locationinformation of a speaking attendee by utilizing an audio positioningalgorithm, and output attendee location information to the remote endthrough network.
 10. The system of claim 1, wherein: the system furthercomprises a black backdrop and a background projector, and thebackground projector works alternately with the panoramic camera undercontrol of the synchronizing unit of the system, or the system furthercomprises a background monitor placed behind the attendee, and thebackground monitor works alternately with the panoramic camera undercontrol of the synchronizing unit.
 11. The system of claim 1, wherein:the system further comprises a background projector, the linearpolarizer is added in front of the background projector and the lens ofthe panoramic camera respectively, and the polarization angle of thepolarizer of the background projector is different from the polarizationangle of the polarizer of the panoramic camera.
 12. The system of claim1, wherein: the system further comprises a background monitor placedbehind the attendee, the linear polarizer is added in front of thebackground monitor and the lens of the panoramic camera respectively,and a polarization angle of a polarizer of the background monitor isdifferent from the polarization angle of the polarizer of the panoramiccamera; or the system further comprises a Liquid Crystal Display (LCD)placed behind the attendee, the linear polarizer is added in front ofthe lens of the panoramic camera, and a polarization angle of lightemitted by the LCD is different from the polarization angle of thepolarizer of the panoramic camera.
 13. The system of claim 1, wherein:the system further comprises a stereoscopic camera and an auxiliarydisplay device, the audio and video communication device transmitsthree-dimensional (3D) videos collected by the stereoscopic camera onthe local end to the auxiliary display device on the remote end throughthe network for displaying.
 14. The system of claim 1, wherein: thesystem further comprises an image correcting/blending unit; the imagecorrecting/blending unit on the local end is configured to performgeometrical correction and edge blending for the images transmitted fromthe remote end; and the images that have undergone the geometricalcorrection and edge blending are displayed through the video displaydevice on the local end.
 15. The system of claim 14, wherein: thedisplay device comprises two or more projectors, the imagecorrecting/blending unit is further configured to split the panoramicimage into images whose quantity is equal to the quantity of theprojectors, and output the split and processed images to the projectorsrespectively for displaying.
 16. The system of claim 1, wherein: theaudio and video communication device receives remote attendee locationinformation transmitted by the audio and video communication device onthe remote end, wherein the location information is detected by theaudio and video communication device on the remote end through the audiocollection device on the remote end; and the audio player reproduces asound field for received audios according to the location information.